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(57)Abstract: 

PROBLEM TO BE SOLVED: To confirm a composition 
result in real time and to improve the usability by an operator 
by providing a means which generates indicator information 
for displaying a composite image, a means which composes 
a panoramic image by using a difference image and a 
movement quantity. 

SOLUTION: A user informs an arithmetic processor 116 of 
transition to a state wherein panoramic composition is 
performed in real time first by using an input device 120. 
The arithmetic processor 116 when entering the panoramic 
photography mode inputs the video signal outputted from a 
video camera part 100, frame by frame, finds the camera 
movement quantities of the inputted frame images, and 
gathers parameters and data needed for the panoramic 

composition. At this time, a the whole or partial area of a display 1 12 is set as a preview screen 
for confirming the composition state and the state of a composite image which is connected in 
timing to the movement of the camera is updated and displayed in real time. 
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* NOTICES * 

Japan Patent Office is not responsible for any 
damages caused by the use of this translation. 

1. This document has been translated by computer. So the translation may not reflect 
the original precisely. 

2 **** s hows the word which can not be translated. 
3. In the drawings, any words are not translated. 



CLAIMS 



[Claim(s)] 

[Claim l] An image synthesizer unit characterized by providing the following A display 
means to display an animation under photography A means to capture a frame image 
one by one from said animation, to compare two frame images with which incorporation 
time of day differs, and to calculate movement magnitude A means to memorize a 
subtraction image and said movement magnitude of said two frame images A means to 
create indicator information which symbolizes a synthetic image compounded from 
memorized movement magnitude and a captured frame image, and a means to 
compound a panorama image using said subtraction image and said movement 
magnitude 

[Claim 2] It is the image synthesizer unit which has a means compound a contraction 
image from a contraction frame image which said indicator information reduced said 
synthetic image, and reduced said captured frame image for a predetermined scale 
factor in an image synthesizer unit according to claim 1, and movement magnitude 
which changed said movement magnitude according to this scale factor, and is 
characterized by for said display means to display said contraction image and said 
panorama image. 

[Claim 3] It is the image synthesizer unit which has a means to create a symbol image 
in which said indicator information is a symbol image which symbolizes said synthetic 
whole image configuration in an image synthesizer unit according to claim 1, and a 
configuration of said synthetic image is shown, and is characterized by said display 
means displaying said symbol image and said panorama image. 

[Claim 4] An image synthesizer unit characterized by setting to an image synthesizer 
unit according to claim 2, and carrying out highlighting of the field corresponding to a 
frame incorporated by the newest in said contraction image. 

[Claim 5] An image synthesizer unit characterized by setting to an image synthesizer 
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unit according to claim 3, and carrying out highlighting of the field corresponding to a 
frame incorporated by the newest in said symbol image. 

[Claim 6] Said subtraction image is an image synthesizer unit characterized by said 
panorama image composition means compounding a boundary between said two frame 
images smoothly in an image synthesizer unit according to claim 1 using this common 
image field also including a common image field of said two frame images. 
[Claim 7] It is the image synthesizer unit characterized by calculating a pixel of this 
expanded subtraction image with interpolation when expanding said subtraction image 
and creating a synthetic image in an image synthesizer unit according to claim 1. 
[Claim 8] An image synthesizer unit characterized by providing the following. A display 
means to display a photoed animation A means to capture a frame image one by one 
from said animation A means to compare two frame images with which incorporated 
time of day differs, and to calculate movement magnitude of an image A means reduce 
this synthetic image remembered to be a means obtain a synthetic image combining a 
frame image captured based on said movement magnitude, a buffer means memorize 
said synthetic image or its contraction image, and a preview display means display one 
field of said synthetic image memorized by said buffer by said buffer when said 
synthetic image or its contraction image exceed a predetermined field, and said one field 
[Claim 9] A record medium with which an image composition method characterized by 
providing the following was recorded and in which a computer readout is possible A step 
which displays an animation under photography A step which captures a frame image 
one by one from said animation, compares two frame images with which incorporation 
time of day differs, and calculates movement magnitude A step which memorizes a 
subtraction image and said movement magnitude of said two frame images A step 
which creates indicator information which symbolizes a synthetic image compounded 
from memorized movement magnitude and a captured frame image, and a step which 
compounds a panorama image using said subtraction image and said movement 
magnitude 

[Claim 10] A step which compares two frame images with which incorporated time of 
day differs, and calculates movement magnitude of an image, A step which obtains a 
synthetic image combining a frame image captured based on this movement magnitude, 
A step which memorizes this synthetic image or its contraction image to a buffer, A step 
which displays one field of this synthetic image memorized by this buffer and which 
indicates by preview, A record medium with which an image composition method by 
which it is having [ when this synthetic image or its contraction image exceeds the field 
range of regular ]*step which reduced and displays this synthetic image and this one 
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field which were memorized by this buffer characterized was recorded and in which a 
computer readout is possible. 



DETAILED DESCRIPTION 



[Detailed Description of the Invention] 
[0001] 

[The technical field to which invention belongs] This invention relates to the image 
synthesizer unit which compounds a panorama image on real time from the dynamic 
image which is applied to the record medium with which the synthesizer unit and the 
synthetic method of compounding the panorama image of a wide-field-of-view angle 
were recorded, especially continues, and this record medium by which synthetic method 
drawing record was carried out. 
[0002] 

[Description of the Prior Art] Two or more images which photoed a different visual field 
are compounded, and the technology which makes the panorama image of the wide field 
of view of one sheet attracts attention. Especially recently, interests have gathered for 
the technology which creates a panorama image similarly from the continuation 
dynamic image photoed while carrying out the pan of the video camera and changing 
the visual field and which is called Video Mosaic. For example, Teodosio of U.S. 
Massachusetts Institute of Technology, L. Bender, W. It depends. Salient Video Stills: 
Content and Context Preserved, Proc. ACM Multimedia 93, pp. 39-46 (1993) etc. - it is 
known. In addition, although various examples of research exist, the panorama image 
has been obtained by sticking an image so that the common area may be found and a 
common area may lap exactly about the image of two or more sheets with which any 
technique was acquired continuously. Since, as for close, the image of a new visual field 
comes to a photography image by following on moving a camera, they are connected and, 
finally the panorama image of the wide field of view is obtained. 

[0003] However, in many examples of research, things for which the precision of a 
synthetic image is raised, such as how such a common area is found with a sufficient 
precision or how to amend a gap of the image periphery by distortion of a lens, are 
thought as important, and it is hard to say that the examination from a viewpoint of the 
usabiliity of the operator using such panorama composition technology is actually 
enough. In order to perform image recognition generally about the natural image which 
cannot specify the class of target image, human being needs to intervene and it is 
necessary to give suitable decision on various aspects of affairs. In this case, the 
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high-speed responsibility of processing influences an operator's user-friendliness 
greatly, depending on a use, it connects that there is no breakdown as an image - even 
having - if it is, it is not few also when the technique excellent in usabiliity which can 
process an input image without delay to the input of real time, i.e., an image, rather, 
and the result can check immediately is more desirable. 
[0004] 

[Problem(s) to be Solved by the Invention] Two kinds of processings need to be 
performed in creation of a panorama image. One is presumed processing of the amount 
of camera motions in which recognize a continuous input image and the coordinate of a 
common area is determined. Another is obtained synthetic processing which moves and 
actually connects an image from an amount. It is most desirable to perform these to 
coincidence on real time moreover from a viewpoint of usabiliity. That is, when the 
shaken amount is calculated, an image ties, and is set automatically and it finishes 
photographing, it is an ideal that the synthetic image is already made and it can check, 
at the same time it shakes a camera. Redo can be done freely any number of times, even 
if it adopts how with missing information to essentially photograph an uncompoundable 
image, on that spot, a mistake is known and the fundamental correction from how to 
photograph can be performed, until the image included in mind is obtained by this. 
[0005] When the shaken amount is calculated, an image ties, and is set automatically, 
and it finishes photographing, and the synthetic image is already made and it enables it 
to check at the same time it shakes a camera, both these two kinds of presumed 
processings and synthetic processing need to process on real time. About presumed 
processing of the amount of camera motions, there is high-speed mode of processing 
shown by JP,11~004398,A "a digital wide camera" by artificers, for example. However, 
about synthetic processing, it cannot be said that even the level which can be performed 
on real time at coincidence fully inquires. After photography, a synthetic result cannot 
be checked but it is inadequate that the amount of motions is just calculable on real 
time to photography and coincidence in the field of usabiliity, if it does not wait for the 
time amount of lamination processing of images, although a calculated part of the 
amount of motions becomes short. 

[0006] The purpose of this invention is to realize high-speed responsibility of the 
synthetic processing which can check a synthetic result on real time in order to raise an 
operator's usabiliity in the method of compounding a panorama image from a 
continuous dynamic image. 
[0007] 

[Means for Solving the Problem] A display means to display an animation under 
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photography in order to attain the above-mentioned purpose, and a means to capture a 
frame image one by one from said animation, to compare two frame images with which 
incorporation time of day differs, and to calculate movement magnitude, A means to 
record a subtraction image and said movement magnitude of said two frame images, A 
means to create indicator information which symbolizes a synthetic image compounded 
from memorized movement magnitude and a captured frame image, and a means to 
compound a panorama image using said subtraction image and said movement 
magnitude are established. 
[0008] 

[Embodiment of the Invention] Hereafter, one example of this invention is explained to 
details. Drawing 1 is an example of the outline block diagram of the system 
configuration for realizing this invention. 100 is the general video camera image pick-up 
section, and transmits the photoed image to a processing unit 116 through the video 
signal line 106. The camera section is connected with a processing unit 116 by the 
control signal line 108, and 116 has come to be able to perform a setup of the camera 
parameter of being as adjusting a white balance **** [, and ], if needed. [ changing the 
zoom scale factor of a video camera ] It can omit, when such a setup is unnecessary. The 
processing unit 116 is the same as the system configuration of the digital computer used 
for the current general-purpose target, especially a personal computer, the display 
output is drawn on the screen of a display 112 through a video signal line, and carrier 
beam directions are notified to a processing unit 116 by the input device 120 from a user. 
Serially, the video signal outputted from the camera section 100 is changed into digital 
image data by the A/D converter of 102, goes into memory 114 through an interface 104, 
and is processed by CPU 110 according to the program stored in another address of 
memory 114. 

[0009] Here, although the signal from a video camera explains in the example in the 
case of analog video format, such as NTSC, when using the video camera of a digital 
output, the A/D converter of 102 is unnecessary. In this case, the converter of image 
format may be instead needed. Moreover, according to the necessity for processing, 
various information can be accumulated in the external information storage device of 
118. 

[00 10] In case various kinds of data created by the processing explained below is stored 
in memory 114 and CPU 110 processes, it is referred to if needed. Although the following 
explanation explains as software which CPU performs altogether, the part or all may be 
replaced and performed by the hardware logic circuit. Moreover, you may be the 
intelligent video camera gestalt which unified the camera section 100, the processing 
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unit 116, and the display 112. 

[001 1] Below, the software flow which is performed on the above-mentioned hardware 
and which solves the technical problem of point ** is explained to details. First, the 
creation method of the panorama image using the above-mentioned software is 
explained. A user tells shifting to the condition (it being called panoramic exposure 
mode below) that panorama composition is first performed on real time to a processing 
unit 116 with the input device 120. When an input device uses a computer system, in 
taking a camera one apparatus gestalt in a keyboard, a mouse, etc., switches, such as a 
"panoramic exposure initiation carbon button", correspond. If a processing unit 116 goes 
into panoramic exposure mode, it will carry out the ** frame input of the video signal 
outputted from a video camera, will calculate the amount of camera motions of the 
inputted frame image, and will collect parameters and data required for panorama 
composition. At this time, that whole or some fields are set up by the display 112 as a 
preview screen for checking a synthetic condition, and the appearance of the synthetic 
image connected according to the motion of a camera is indicated by updating on real 
time at it. 

[0012] The image is shown in the display 112 of drawing 1 . In this invention, the 
synthetic image 122 (it is hereafter called a preview image) for a preview display is 
characterized by always carrying out a scaling and displaying that it becomes the 
greatest magnitude in which the whole fits in a preview screen. By this, the grasp at an 
instant of the whole synthetic condition is enabled. This is fill uped with the suitable 
background color, although the null portion 124 in which an image does not exist exists 
with the configuration of a synthetic image when a preview screen is a rectangle at this 
time. The location of the newest input frame in a synthetic image is surrounding, or 
changing a color and carrying out highlighting by the frame, and the role of the compass 
which does not puzzle the direction from which a camera should be moved is also given 
(126). 

[0013] Again, with the input device 120, if a user directs termination in panoramic 
exposure mode, he stops a new frame image input and will be in a standby condition. 
The preview image updated by the last frame input by the display 112 is held, and a 
user looks at the image, judges whether the result of a request is obtained, if good, he 
creates the synthetic image (it is hereafter called a final output composition image) as a 
final output of the real size which is not an object for a preview, and saves at the 
external information recording device 118 etc. 

[0014] An example of a flow chart which realizes the above-mentioned function to 
drawing 2 is shown. This algorithm makes the following three points the basic concept 
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in order to enable high-speed processing indispensable to the above-mentioned function. 
[0015] (1) Create the synthetic image of contraction size only to previews apart from a 
final output composition image. 

[0016] (2) Let the image memorized in the last composition images be only an 
indispensable field. 

[0017] (3) Reconcile creation of the preview image of an arbitration configuration and 
arbitration size, and memory effectiveness. 

[0018] (l) is a device for reducing the time amount concerning synthetic processing. It is 
not necessary to be the image of size with the created image not necessarily big 
generally, in order to judge whether it is a desired thing. So, at the time of photography, 
a final output composition image is not created but creates the synthetic image of 
contraction size separately as an object for a preview. It is made for a play view image to 
be mostly completed to coincidence with photography by this. Since composition is 
performed based on the camera motion estimate from which this preview image was 
obtained, near, distortion, a mistake, etc. are extremely reflected in an actual synthetic 
result as it is. If a user judges whether the desired image was obtained and is checked 
by this preview with right **, he will create the final output composition image of actual 
magnitude anew. Since it can redo immediately and the time amount of composition of 
real size can be saved when it is judged as failure, an efficient activity can be performed. 
[0019] (2) is a device for reducing the chart lasting time of the former data for a final 
output composition image. When (l) is adopted and a final output composition image is 
made to be created later, it is necessary to record temporarily the REMU image which is 
the former data. It becomes the inhibition factor of the high-speed processing by it being 
as that required memory space ****** s **** [, and ] to record the new frame image 
inputted one after another one after another. [ that it takes a data transfer time for a 
long time ] 

[0020] Here, as shown in drawing 3 , a panorama composition image piles up every 
sheet of a continuous frame image, and is obtained. Drawing 3 is an example which 
creates a panorama image from a series of panimages, expresses typically the image 
which consists of a frame image with which the bottom continues, and shows the 
panorama image with which the bottom is compounded from it. Two or more lines which 
tie up the upper and lower sides express the correspondence relation of frame images. 
By the lower panorama image, the continuous field (it is hereafter called a common 
area) which is in agreement always in image by inter-frame exists, and redundancy is 
very high as data. Then, a panorama image is fully reproducible by combining only the 
field of the portion which excluded the this continuation inter-frame common area. 
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Therefore, if only the field which excluded this common area per frame is recorded, the 
amount of data which should record per frame is reduced sharply, a data transfer time 
will be shortened and its required memory space will also decrease. Although the whole 
image is needed only for superposition of the last frame, since it is only the 1 time 
uniquely, and it is the last and the following frame does not exist even if time amount 
cuts some in many to data transfer, the problem of not being of use for the processing for 
degree frame is not generated, either. 

[0021] Although it is about (3) of the last, a preview image conforms to a motion of a 
camera out of photography further, it is not only obtained by photography termination 
and coincidence, but is updated synchronizing with a motion of real time, i.e., a camera, 
and a check is [ way ] easy for a user and it is user-friendly. However, the panorama 
image compounded becomes large in monotone as it moves a camera, and the 
configuration and magnitude become various depending on how to move a camera. In 
order to display a preview image, the panorama image compounded for a preview must 
always be created on the memory area, but in order to create the possible preview image 
of all configurations and magnitude on a memory area, a beforehand very large memory 
area must be taken. 

[0022] Though a contraction frame image is used with the indicator of (l), the capacity 
of a required memory area is huge. And there will also be many memory portions which 
are not used in fact, and, as for an emergency, the utilization ratio of memory will 
become low. Of course, the direction from which a camera is moved first can be 
determined and only a part required for it can secure a memory area. However, by this 
method, even if the complicatedness which must show the direction from which a user 
moves a camera beforehand arises, furthermore mind changes a user on the way, it 
cannot respond at all. Moreover, finally it becomes the image of the magnitude of which, 
or a user cannot predict in many cases. To correspond to the synthetic image of 
unrestricted size and an unrestricted configuration is desired. 

[0023] Here, the magnitude of the preview screen for displaying a preview image on a 
monitor is immobilization, and when the magnitude of a preview image becomes larger 
than the preview screen of the magnitude of immobilization, a contraction scaling is 
carried out and it is displayed to suit the screen size. Conversely, if it says, the number 
of pixels exceeding the resolution of a preview screen will become useless in case a 
preview image is displayed. 

[0024] Then, the magnitude of a preview image is n times (length or ratio as horizontal 
length.) of the magnitude of a preview screen. When nxn times are exceeded in area, the 
magnitude of a preview image is compulsorily reduced by 1/m time (similarly in area 
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l/(mxm) twice). And when the magnitude of the reduced preview image exceeds n times 
of a preview screen again, it repeats as 1/m time contraction of a preview image is 
performed. The effect that the preview image which can restrict the memory area size 
for preview images to n or less times of the size of a preview screen, and is displayed on 
a preview screen by this does not have deterioration of resolution compared with the 
case where it has a memory area beyond it is acquired. 

[0025] Based on the above thing, it returns to explanation of the flow chart of drawing 2 . 
Various kinds of initialization is first performed by processing 200. For example, the 
memory area (it is hereafter called a preview buffer) of the magnitude which carried out 
magnitude of a preview screen in all directions [ n times as many as this / each ] is 
secured as a working buffer for preview image creation. A preview image is compounded 
in this preview buffer, the field called a preview viewing area is set to a preview buffer, 
and this field fills the magnitude of a preview screen exactly - as - always - a scaling - 
that is, zooming is carried out and it becomes the display of a preview screen. 
[0026] A sequential preview viewing area is expanded so that the whole preview image 
may always be contained, as the field of the same magnitude as the preview screen of a 
buffer center section is set up as a preview viewing area at first and a preview image 
becomes large on a preview buffer. Subsequently, the initial value of the preview 
reduction percentage r which is a variable for preview image creation is calculated (202). 
r is the ratio of the magnitude of the frame image inputted from a camera, and the 
magnitude of a preview screen, and is a parameter adjusted so that an input frame 
image may be displayed to the limit of a preview screen in an initial state without a 
motion of a camera. For example, it is r= 1 when magnitude of an input frame image 
and magnitude of a preview screen are made the same. It is set to 0. 
[0027] In the continuing processing 204, the frame image of the one newest coma of the 
image photoed with the video camera is inputted. And movement magnitude detection 
processing of an image is performed, comparing between the frame images inputted one 
after another. Although this movement magnitude detection considers various methods, 
such as an optical flow, the method indicated by the "digital wide camera" of Japanese 
Patent Application No. 9*153303 is used for it as a method computable at a high speed 
here. This method enables high-speed processing very much, in order to calculate 
movement magnitude from projection distribution of an image. 

[0028] In processing 206, the level projection distribution Px and the perpendicular 
projection distribution Py are created from this frame image. Level projection 
distribution is what took the average of the color and brightness about each line of the 
pixel which constitutes an image, and is the single dimension sequence of the average 
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only for a line count. Perpendicular projection distribution is what took the average of a 
color or brightness similarly about each train of an image, and is the single dimension 
sequence of the average only for the number of trains. For example, the list of the value 
of perpendicular projection distribution has the property shifted to the forward direction 
or hard flow corresponding to it, when an image moves to right and left. Perpendicular 
projection distribution is similarly shifted to order hard flow corresponding to vertical 
movement of an image. Therefore, the horizontal / perpendicular projection distribution 
created from a certain frame image are compared with the horizontal / perpendicular 
projection distribution created from the frame image inputted into the degree, and if it 
asks for which the sequence of the average shifted, respectively, the movement 
magnitude of level/perpendicular direction between the images of two sheets can be 
found. Although the list of the value of projection distribution is completely in 
agreement with neither a noise nor some deformation since it is image change, one of 
the two of two projection distribution to compare is shifted by delta pixel one by one, and 
is compared (all integers for which delta takes the range of -R to +R.). R obtains the 
value which shows the retrieval range set up beforehand, and delta when dissimilarity 
becomes low most as movement magnitude. Dissimilarity uses absolute total of the 
difference of the color of the pixel of a corresponding line or a train, or each average of 
brightness etc. (210-212). However, since the frame image in comparison with the past 
does not exist in the first image input, it returns to processing 204 (208). 
[0029] In processing 214, from obtained level / the amount delta of vertical migration, it 
asks for the common area of the last frame image and the present frame image, and 
records only about the field of portions other than the common area in the last frame. 
By this, there is little storage capacity compared with the case where it records about all 
fields, and it ends. 

[0030] Moreover, in processing 216, the movement magnitude for preview composition 
which multiplied the value of the obtained horizontal / the amount delta of vertical 
migration by r is calculated, and the image reduced by r times is drawn in piles in the 
location which shifted only deltaxr obtained above to the contraction frame image which 
drew on the preview buffer immediately before. Thus, the preview viewing area updated 
so that the expanded whole preview image might be included is displayed on a preview 
screen (218). 

[0031] At this time, when the whole preview image is as Hasama to the limit of the edge 
of a preview buffer, (220) and the whole preview buffer are reduced by 1/m time (222). 
At this time, a preview viewing area is similarly reduced by 1/m time. And r multiplies 
by 1/m similarly. 
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[0032] The typical flow of contraction processing of this preview buffer is shown in 
drawing 4 . 400 shows the whole preview buffer and 402 is a preview viewing area. 404 
shows a preview image. A location and magnitude change so that a preview viewing 
area may always be circumscribed to the whole preview image. A expresses the 
situation immediately after panoramic exposure initiation, and the inputted frame 
image is drawn by the center section. The preview image is extended, also follows a 
preview viewing area, and becomes large as a camera is moved (B). And since a buffer 
will be overflowed if a preview image reaches to the edge of a preview buffer as shown in 
C, the whole preview buffer is reduced by 1/m time here (D). At this time, a preview 
viewing area is also reduced for the same scale factor. And if a preview image is 
extended further, drawing of an image will be continued according to it. 
[0033] Here, if n= 3 and m= 2, first, the center of a preview buffer and the center of a 
viewing area will be doubled, and the size of the preview viewing area when arriving at 
elongation and an edge after that according to a photography image will become always 
larger than the twice (a length ratio in every direction is twice) of the original size of a 
preview screen, therefore, this - every the whole preview buffer 1/2 twice (it is a length 
ratio in every direction 1/2) - even if - the size of a preview viewing area is not less than 
the original size of a preview screen In addition, when a preview viewing area reaches 
that of a buffer exactly at the time of n= 3 and m= 2, a preview viewing area becomes 
twice the original size of a preview screen. Then, if a preview buffer is doubled 1/2, the 
size of a preview viewing area will become the same as the original size of a preview 
screen. 

[0034] The result which hardly changes to the case where secure a huge memory area, 
draw a preview image freely, and it thins out and displays on the size of a preview 
screen on it by this will be obtained only by preparing n times as much memory space as 
a preview screen. Moreover, since what is necessary is just to thin out at intervals of a 
pixel simply in the case of m= 2, contraction can also be extremely performed at a high 
speed. 

[0035] Here, it considers creating a panorama image 10 times the magnitude for 
example, of an input frame image. By the method of securing a memory area beforehand, 
without being based on the above-mentioned method, in order to preview the panorama 
image of this area, an about 400 times as many memory area as an input frame image is 
needed beforehand. Since this cannot predict in which direction a camera moves from 
the beginning, it is because it is necessary to secure the rectangle field which has one 10 
times the length of this in each vertical and horizontal direction centering on the first 
input frame image, respectively. However, according to the method of this invention, it 
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is [ at one at most 9 times the magnitude of this ] good in the case of n= 3. Since this 
invention guarantees unrestricted size by one 9 times this magnitude of this, the 
difference of a memory area with the conventional method is followed on the magnitude 
of the panorama image to compound increasing 20 times and 30 times, and becomes 
large exponentially. 

[0036] Furthermore, in high resolving still picture composition processing in which the 
zoom image was used, the above-mentioned technique has an effect further. The 
principle of the high resolving still picture composition by the zoom is shown in drawing 
5 . This fundamental view is already mentioned reference Salient Video Stills. : It is 
shown in Content and Context Preserved. A big high resolution image can be obtained 
by making the frame image when zooming in most into criteria size by piling up other 
frame images according to the rate of a zoom. This high resolution image becomes 
highly minute little by little gradually as a periphery approaches a core, although a 
pixel only becomes large simply. The periphery is fading and its image photoed with the 
usual stiU camera is also close to the property of the high resolution image by which 
superposition composition was carried out, when a focus is doubled with a center. That 
is, an image equivalent to having copied using the image sensor of expensive high 
resolution will be obtained using the image sensor of a cheap low resolution. 
[0037] However, by such conventional technique, the expansion processing time also 
becomes long and a real-time operation becomes difficult further rather than the time of 
panorama composition as superposition processing must be performed and a dilation 
ratio becomes large, after expanding the image at the time of zoom out. What is 
necessary is just to draw the first frame by actual size, and to pile up the rest, 
expanding a frame image, if a huge memory area can be prepared first in the case of an 
image which carries out zoom out in monotone still gradually. However, in the case of 
an image which zooms in reverse, it is necessary to expand the synthetic image, and 
time amount requires expanding a huge memory area the whole ** frame very much so 
that the frame image which is the purpose of high resolving image composition and into 
which resolution was inputted at the end by composition with slight height may always 
become actual size. This is the technical problem of the real-time operation no one but 
which cannot predict any motion [ what kind of camera ] there is beforehand. 
[0038] On the other hand, if the view of this invention is applied, a preview can be 
checked on real time in few memory areas. About the technique of asking a high speed 
for the rate of a zoom, the technique already shown in Japanese Patent Application No. 
10-53100 by artificers "a super resolution camera" can be used, for example. When it 
checks here, it is piling up the frame image inputted, adjusting size so that it may 
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become the contraction scale as composition of a high resolving image with the same 
common area between continuous frame images. Therefore, like the time of panorama 
composition, in order to obtain a final synthetic image, a common area requires only the 
character type outer frame portion [ in / in the case of the image of zoonrin / the last 
frame ] of RO except a common area that what is necessary is to record only once. 
Moreover, what is necessary is just to merely draw the frame image reduced according 
to the rate of a zoom one after another to the preview buffer in creation of a preview 
image. 

[0039] The rate of a zoom is defined here as a ratio when carrying out size doubling so 
that a common area may be in agreement on the basis of the frame image inputted first. 
It is 1 when there is no zoom. In 0 and zoom-in, it is smaller than 1, and, in the case of 
zoom out, it becomes large from 1. On the other hand, in the case of the image of zoom 
out, since the range where the newest frame is larger is copied, the character type outer 
frame portion of RO except the common area in the newest frame is recorded shortly. If 
creation of a preview image hits, this outer frame portion is reduced according to the 
rate of a zoom, and it draws. If the rate of a zoom becomes [ the size of a preview image ] 
large, since the size of a preview buffer will be exceeded in the case of zoom out, the 
whole preview buffer is reduced to 1/m like the time of panorama composition in that 
case. Thus, an expansion required for preview image creation also in the case of a zoom 
is only processing to which only the character type outer frame portion of RO is 
expanded to n or less times, and can be clearly processed at a high speed. By the above, 
the result of high resolving image composition can also be checked on real time to 
photography and coincidence. 

[0040] An example of the former data recorded in this example for a final synthetic 
image is shown in drawing 6 and drawing 7 . Data is divided roughly into the panorama 
file structure object 600 and the panorama clip structure 700. The former panorama file 
structure object is a header which gathers the whole former data, and the latter 
panorama clip structure is image data as which record is determined for every frame. 
The real size of a file ID 602 and a synthetic image is stored in a panorama file structure 
object in order of width of face (604) and height (606) as an identifier which shows that 
consecutiveness data is former data for image composition. Since the final size of a 
synthetic image can be expressed as the minimum rectangle circumscribed to it even if 
it was determined from the amount of camera motions and has taken the special 
configuration, only width of face and height are stored. Moreover, the color number 608 
(it is one full color 24 bits etc. about whether they are for example, 256 colors) and the 
graphics format 610 (for example, it is the YUV format about whether it is RGB format) 
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[0041] Moreover, the starting point coordinate of the rectangle field circumscribed to a 
synthetic image is stored as the panorama image x and y offset (612 614), and the 
coordinate of the panorama clip later mentioned on the basis of this coordinate is 
calculated. The panorama clip structure 700 is located in a line as data following this 
header information. The panorama clip structure serves as data for every one frame, 
and stores the location and size of the frame image in a synthetic image. First, 702 is 
the total data size and stores the amount of image data. The image starting point 
x-coordinate 702 expresses the x-coordinate value of the starting point of the frame 
image matched with this panorama clip structure when making into the starting point 
coordinate of a synthetic image the coordinate shown by the panorama image x 
above-mentioned offset 612. Similarly, the image starting point ycoordinate 704 
expresses the ycoordinate value of the starting point of a frame image. The image 
width of face 706 and the image high 708 express real [ that a frame image is inputted ] 
size. The size of the frame image on a synthetic image is calculated by multiplying this 
by the rate 710 of a zoom. 714-720 specify the range of the above-mentioned common 
area in this frame image. The common area starting point x-coordinate 714 and the 
common area starting point y-coordinate 716 express the starting point coordinate of 
the common area in a frame, are the common area width of face 718 and common area 
high [ 720 ], and express the magnitude. All pixel data is located in a line with the pixel 
data 722 in order, scanning in the direction of the lower right from the upper left. When 
the suitable value for 714-720 is in close, the pixel data about a common area is 
disregarded simply. 

[0042] Next, how to create a final synthetic image from the former data memorized 
above is explained. With reference to the panorama file structure object 600, the width 
of face and the height of a synthetic image are investigated first, and the rectangle 
image field of the magnitude is secured in a memory area. And with reference to the 
panorama clip structure 700 which follows, the location and magnitude which should 
draw are investigated, it asks for the relative position in the secured memory area, and 
image data 722 is written in. However, when at least one clip whose rate of a zoom is 
less than 1 time exists, proportionality conversion of the rate of a zoom of all the 
panorama clip structures is carried out so that the smallest rate of a zoom may be first 
set to 1, namely, so that the frame image which zoomed in most may be drawn by actual 
size. In this case, the size of a synthetic image also becomes large proportionally. It is 
lost by this that a frame image is reduced and composition is performed only in the 
direction in which resolution is raised. 
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[0043] Smoothing processing can be performed in order to show more beautifully the 
image composition described above. Generally, when sticking an image, there may be 
distortion in the time of the input image according to the aberration of a lens in that the 
amount of motions cannot be found correctly ****, or the brightness in a photography 
time may be changed, and a delicate boundary line may occur in the knot of an image. 
With human being's perception, this boundary line needs to amend in order to give an 
unpleasant impression. The easiest method of performing such amendment is sticking 
so that an edge's left for applying paste may be given among the images to stick, a 
boundary line's may be inserted and it may change to the color of the image of another 
side gradually from the color of one image. The image which was simplified very much 
and showed the concept is shown in drawing 8 . 800 shows signs that the image of two 
sheets is stuck simply. Here, in order to simplify explanation, the image of two sheets 
assumes that it is uniform by specific brightness, respectively. On the boundary line of 
an image, if change of the horizontal brightness at this time is looked at, as shown in a 
graph 802, it will change in an instant and this will form a boundary line 808. On the 
other hand, as shown in 804 and 806, if it sticks so that it may become a gradual change 
from the brightness of one image to the brightness of the image of another side, a 
boundary line will not be conspicuous [ an edge left for applying paste is given to the 
boundary of an image, and ]. 

[0044] In order to realize the above-mentioned smoothing processing, when recording 
former data by the synthetic method of this invention, only the part of an edge left for 
applying paste should record a larger field. By the increment in the necessary minimum 
amount of data, smoothing becomes possible by this. And the synthetic image connected 
smoothly is obtained by performing smoothing processing about the part of an edge left 
for applying paste in the case of next synthetic processing. The field made into an edge 
left for applying paste is chosen in the form which narrows the common area between 
the continuous images of two sheets fundamentally. The example in the case of having 
arranged so that a common area may lap with drawing 9 about the two frame images 
900 and 902 is shown. Supposing 902 is a new frame image in time, the L character 
mold hatching field 904 at the upper left of drawing will be recorded as former image 
data. Since an edge left for applying paste should just bury the boundary line of two 
images gradually, it serves as a field which needs the field which swelled width of face 
for the direction which narrows a common area in accordance with a boundary line. In 
the example of drawing 9 , the field used as an edge left for applying paste turns into the 
hatching field 906 of the L character mold inside 904. It is shown in a list at drawing 10 
by carrying out combination of the typical amount of camera motions, and the 
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edge-left-for-applying-paste field at that time, deltax and delta y show the movement 
magnitude of level/perpendicular direction among drawing, respectively. The pattern of 
the same edge-left-for-applying-paste field in case a zoom is further shown in drawing 
11 is shown. By a diagram, the case of zoom-in is shown. Thus, even if there is a zoom, 
from the boundary line between the images when doubling magnitude so that the 
magnitude of a common area may become the same, it extends in the direction of a 
common area, and an edge-left-for-applying-paste field is decided. Also in zoom out, it is 
the same only by the relation to reverse of the frame images 900 and 902 becoming. 
[0045] In addition, when creating a highly minute image, in case image data is stuck 
according to the rate of a zoom, it cannot be overemphasized that a still better-looking 
image is obtained by expanding using pixel interpolation technology, such as bilinear 
interpolation instead of expansion of a simple pixel. And a still more beautiful image is 
obtained by performing the above-mentioned smoothing processing for the image data 
expanded with pixel interpolation. 

[0046] Moreover, although the technique of creating and displaying a preview image on 
real time was described above, as it indicates by simple instead of a preview image as 
the symbol which shows the situation under composition, coexistence with usabiliity 
and much more improvement in the speed can also be aimed at. For example, since it is 
only a line drawing, drawing becomes possible at a high speed, at the same time 
necessity is also lost by reducing and sticking a frame image by showing with a line 
drawing whether not the preview image itself but the silhouette, i.e., a synthetic image, 
is made of what kind of configuration. The example of a preview screen based on this is 
shown in drawing 12. On a preview screen, an input image is displayed as it is, and an 
indicator indication also of the line drawing 1200 is given together with it on it. 
Moreover, in order to show where [ in a synthetic image ] the frame under current input 
hits, as shown in 1202, highlighting of the field which a frame image current in the 
inside of a line drawing occupies may be carried out. The purpose of checking a 
synthetic condition on real time is realized more by the operation engine performance of 
low cost by this. 
[0047] 

[Effect of the Invention] According to this invention, in creation of a panorama image or 
a high resolving image, the outstanding usabiliity which can be retaken can be freely 
offered any number of times until the image which could check the advance condition of 
processing and the result to photography and coincidence, and went into them at mind 
is obtained. 
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DESCRIPTION OF DRAWINGS 



[Brief Description of the Drawings] 

[Drawing l] It is an example of system configuration drawing for realizing this 
invention. 

[Drawing 21 It is an example of the flow chart of the processing for realizing this 
invention. 

[Drawing 31 It is image drawing which compounds a panorama image from the 
animation which carries out a pan. 

[Drawing 41 It is an example of the contents of the preview buffer at the time of 
performing this invention. 

[Drawing 5] It is image drawing which compounds a high resolving image from the 
animation which carries out a zoom. 

[Drawing 61 It is an example of the data structure used by processing of this invention. 

[Drawing 7] It is an example of the data structure used by processing of this invention. 

[Drawing 8] It is image drawing showing the image lamination method. 

[Drawing 9] It is drawing explaining each image field which processing of this invention 

defined. 

[Drawing 101 It is an example of the pattern of an edge-leftfor-applying-paste field used 
by processing of this invention. 

[Drawing 1 ll It is an example of the pattern of an edge-left-for-applying-paste field used 
by processing of this invention. 

[Drawing 121 It is an example of an indicator display used by processing of this 
invention. 

[Description of Notations] 

100 [ - A video signal line, 108 / - A control signal line, 110 / - CPU, 112 / - A display, 
114 / - Memory, 116 / " A processing unit, 118 / - An external information storage 
device, 120 / Input device. ] - The camera section, 102 - An A/D converter, 104 - An 
interface, 106 
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